SRE internships: a practical reliability project

Define “working” before collecting metrics

For this original exercise, a good request must succeed within 300 milliseconds. Set a target of at least 99.5% good requests for the defined test batch. These are project choices, not a recommended target for every service or a promise about production availability.

Create a small local endpoint and record each request’s outcome and elapsed time. Generate bounded traffic only against that environment. Add a deliberately slow dependency in the test harness so you can distinguish a successful response from a response useful within the time limit.

A successful response can still violate the promise

Suppose the invented 1,000-request batch has the following outcomes. Every request appears in exactly one row.

• Success alone is 990 ÷ 1,000 = 99%. The defined good-request rate is 980 ÷ 1,000 = 98%. The first number hides ten slow responses.
• A 99.5% target allows five bad requests in this batch. There are 20, or 15 more than allowed. This batch does not establish a monthly availability result.
• After a fix, rerun the same scenario and report all outcomes. Reducing visible errors by waiting much longer can improve one metric while leaving the user’s problem unresolved.

Explain the recovery and its trade-off

Write a short incident note with the first bad observation, the relevant test evidence, the recovery action and the check showing what changed. Keep a failed case alongside a normal case so that the fix cannot simply hide the problem from the measurement.

Retries are a useful follow-up question. If 20 original requests each make an initial attempt plus three retries, they can produce 80 attempts. State the retry limit and the conditions that stop retrying; more attempts are not free recovery. Test this only in your controlled environment.

A portfolio can show the defined metric, the slow-dependency fixture and the before-and-after request records. Call it a local experiment, not production-scale reliability. Ask the employer who reviews operational changes and what rollback or safe-test route an intern uses.

Before you apply

Eligibility

Read the role’s programming and systems requirements. Do not assume that prior independent production on-call experience is mandatory.

Timing

Datadog describes summer internships of 12 or more weeks and co-ops lasting four to six months.

Location

Check the office and any after-hours expectations, including supervision and whether operational duties are part of the role.

Pay

Calculate pay from confirmed hours and duration. Clarify unusual scheduling terms before accepting.

Sources: Datadog early careers

Job titles to search

Site reliability engineering intern

Improve how reliably a service meets a defined user-facing target.

Infrastructure or production engineering intern

Build deployment, capacity or operational tooling.

Observability engineering intern

Make useful system behavior visible through measurements and investigation tools.

Where to look

Search routes, not confirmed vacancies.

Compare the work and try an example

Use a role comparison or worked exercise to choose your preparation before tailoring an application.

• Internship skills: distinguish required, preferred and taught on the job
• Build an economic consulting work sample that another person can reproduce
• Citadel versus Citadel Securities internships: choose the employer and role