A New Book on Site Reliability Engineering (SRE)!

I just finished 15 months of primary 24x7 on call. Although it is always stressful to be paged in the middle of the night or on a weekend or holiday I was able to lean on my SRE background to ensure that every alert that woke me up faithfully indicated…

Queues are commonly used to lineup work to be done. Consider a queue at a store. The first person in line is the first to be serviced. This is referred to as a “first in first out queue” (FIFO). This post details an implementation of a FIFO queue using a…

Data operations is the act of keeping data pipeline operational, and the data flowing. Data operational maturity focuses on answering: Is a pipeline running? Is data fresh and consistent? Is the data correct? This data operational maturity model draws heavily from Site Reliability Engineering (SRE). This data operational maturity model…

Large data deployments often contain multiple levels of derived tables. Understanding lineage is critical for resolving data downtime and bugs. Often table hierarchies grow out of control with time, which makes maintaining documentation and debugging data issues time intensive and costly. This post describes a method for deriving data lineage…

TL;DR I hit the 1024 File descriptor limit in lambda. AWS has a bug handling this case, in which END and REPORT logs were omitted. The fixed required bounding File Descriptor usage by configuring the go http client (*http.Client). Background I recently worked on a project to extend application log retention…

Views are an important tool in legacy warehouse environments. Views act as an interface which decouples clients (like tableau, looker, applications, etc) from underlying data sources. Enabling backwards compatible changes is especially important in legacy environments, where the full scope of clients may be unknown. …

Snowflake does not currently support sub query pruning which can have serious cost implications for DBT incremental updates against external tables. Careful care must be taken When using DBT incremental models to query against large partitioned external tables. …

Probing is a technique to perform regular checks on a service using a short interval. Probes provide signals that can significantly cut down debug time. …