addressalign-toparrow-leftarrow-leftarrow-right-10x10arrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscontroller-playcredit-cardcrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobe--smallglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1languagelaunch-new-window--smalllight-bulblinklocation-pinlockm-swarmSearchmailmediummessagesminusmobilemoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonprintShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahooyoutube

Re: CTO School list: "Big Data Operations - Data Eng SRE ?"

From: Sean H.
Sent on: Monday, May 27, 2019 9:12 PM

Hi Jean,

Data & ops is my practice...  I can definitely help.  

I like to say that since I am a *small fish* (one guy) you as a client would be a *big fish* for me.  That means you get a lot of handholding, attention, ownership, and experience behind the work I do.  

Also I've worked for many CTOs here.  Please ask around...


PS:  I blog a lot on related topics...

Is Data your dirty little secret?

What are data spot checks?

How to build an operational datastore on AWS with S3 & Redshift

On Wed, May 22, 2019 at 11:22 PM Jean Barmash (Meetup) <[address removed]> wrote:
Jean Barmash (Organizer) sent a message to the CTO School mailing list
Big Data Operations - Data Eng SRE ?

My current company does a lot of Big Data / Data Science work. In our data engineering, some teams have a large percentage (i.e. 30-40%) of operational work - i.e. things like running jobs, loading data, etc. Things that need to get done but are not ultimately value generating in the same was as improvement are.

We tried to create roles for operations and hired junior engineers right from out of college. , but after some initial time, they want to be engineers and get bored with too much operational work.  The good news is that we hired good engineers, but the bad news is that they are unhappy if they do too much non-coding work.

One of the solutions we are considering (combining some of the below) is introducing equivalent of SRE but for data operations. Their goal would be to both operate and improve the pipelines over time. There is both a question of viability, operating model (centralized or embedded in nest), and what type of people you hire for this role.

Has anybody tried something like this? Would love to learn from your experience, especially about the hiring profile for this type of role.



P.S. outside of the SRE idea, I can think of the following solutions:
  • Automate more things - this is a long term solution, I am not sure we have the ROI to justify this. We have already automated a bunch of stuff in some areas.
  • Hire lower-cost people who enjoy more routine work and are not trying to become engineers. Would love to know if somebody tried this approach.
  • Rotate the work as fairly as you can so that engineers feel there is some fairness.  Note - this does not solve the root cause of the problem.

Jean Barmash (LinkedIn)
Co-Founder & Organizer,  CTO School Meetup
VP Engineering, Komodo Health - Big Data / ML Products.  Hiring for many roles in NY & SF.

Sean Hull :: Devops. Data. Innovation.
m: [masked]

People in this
group are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy