Repository navigation
CoP: Data Science: Create Data Ops Tutorial #154
Description
Overview
We need to have a Data Ops Tutorial page that will answer any q
Action Items
- Create a Google Doc in the folder provided under resources
- Draft an introductory paragraph explaining what the tutorial resources cover and why a new data scientist would use them for working with data at Hack For LA
- Identify resources with vetted tutorials covering important skills within the tutorial area, adding to the draft
- Write Clear Learning Objectives
- State what the learner will be able to do by the end of the tutorial (e.g., “Set up a basic DataOps workflow with GitHub and Google Sheets”).
- Write Prerequisites
- Required skills (e.g., basic Python, Git/GitHub familiarity).
- Required tools/accounts (e.g., GitHub, Google Drive access, Looker viewer permissions).
- Write Setup Instructions
- Links to repos, sample datasets, or starter scripts.
- Environment setup (e.g., clone repo, install dependencies).
- Write Step-by-Step Walkthrough
- Guided example showing one DataOps pipeline end-to-end.
- Screenshots or code snippets illustrating key steps.
- Write Hands-On Exercise
- A small project/task learners can replicate (e.g., build a lightweight validation script, update a dashboard, or run a data ingestion pipeline).
- Write Common Pitfalls & Troubleshooting
- Document frequent mistakes (e.g., schema mismatch, GitHub permissions).
- Tips for debugging errors in automation or dashboards.
- Write Discussion / Reflection
- Prompts for learners to connect the tutorial to civic tech use cases.
- Questions like: “How would you adapt this pipeline for a different dataset?”
- Write Next Steps / Advanced Topics
- Suggested follow-ups (e.g., CI/CD for ETL pipelines, adding alerting, scaling beyond Google Sheets).
- Write Glossary / Key Concepts
- Definitions of terms like ETL, schema governance, human-in-the-loop.
- Add additional content to References & Resources
- link to relevant Hack for LA repos or dashboards (where possible).
- Review the draft with the Data Science CoP
- Add to the wiki page
Resources/Instructions
Wiki page
Location for any files you might need to upload (drafts, images, etc.)
Tools that are core that should be mentioned:
- EC2
- Lambda
- RDS
- Athena/Hive
- Flask
Examples of resources that would be useful to include:
- Web how-to/tutorial/walk-throughs
- Youtube playlists or videos demonstrating tools
- Links to blogs or platforms with subject matter experts
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
SANJCA commentedon Apr 2, 2022
[-]Create Data Ops Tutorial[/-][+]CoP: Data Science: Create Data Ops Tutorial[/+]28 remaining items
akiokaki commentedon Aug 29, 2025
Instructions
chinaexpert1 commentedon Aug 30, 2025
chinaexpert1 commentedon Aug 30, 2025
ExperimentsInHonesty commentedon Sep 9, 2025
akiokaki commentedon Sep 10, 2025
chinaexpert1 commentedon Oct 14, 2025
chinaexpert1 commentedon Oct 21, 2025
chinaexpert1 commentedon Nov 11, 2025
unassigned @akiokaki he ghosted us.
great-wave commentedon Oct 26, 2026
Progress: I've completed the tutorial except for the hands-on exercise. The tutorial is forking the district-files and recreating the pipeline created there (changing the forked repo to change from google drive upload to github action artifact.) This is due to an inability to add google cloud service accounts to the data science google drive account.
Blockers: Do we want to commit to a data-ops workflow of Python, Github Actions, Google Drive for storage? None of the services listed in the issue (EC2, Lambda, RDS, Athena/Hive, Flask) are free and open source. Neither is Looker Studio. Do we want to standardize the use of Python 3.10 as used in district-files, and uv as the python installer for virtual environments? Google drive does not allow for individuals to add their self-created service accounts to the data science google drive account, so they cannot upload the data and logs to google drive. Is a github artifact ok for now?
Availability: 6hrs
ETA: Could be finished this upcoming week, depending on schedule for decision making on tool standardization.
Pictures: https://docs.google.com/document/d/15WdOrTtKGuZL3rPuQm3h4SEWuUL4vML0XtPas8fBIwc/edit?tab=t.0#heading=h.3ry73kog7dqg
chinaexpert1 commentedon Oct 27, 2026
Great work. Especially the practical example. Please add a discussion of principles, as per the following:
| Principle | Description
-- | -- | --
1 | Automation First | Automate ingestion, validation, transformation, testing, deployment, and monitoring to minimize manual intervention.
2 | Data Quality by Design | Build quality checks into pipelines rather than inspecting data only after processing.
3 | Continuous Integration / Continuous Delivery (CI/CD) | Version, test, and deploy data pipelines using automated engineering workflows.
4 | Reproducibility | Ensure that identical inputs, code, configurations, and execution environments produce consistent outputs.
5 | Data Observability | Continuously monitor freshness, completeness, volume, distribution changes, schema changes, and pipeline failures.
6 | Data Lineage and Traceability | Track the origins of data, its transformations, and dependencies between upstream and downstream systems.
7 | Version Control | Track changes to pipeline code, schemas, configurations, and datasets or dataset references.
8 | Collaboration and Shared Ownership | Encourage data engineers, analysts, scientists, and operations teams to work together using common standards.
9 | Incremental Development | Deliver small, frequent improvements rather than large, infrequent pipeline changes.
10 | Orchestration and Dependency Management | Coordinate data processing tasks, execution order, retries, scheduling, and recovery from failures.
11 | Security and Governance | Apply access controls, encryption, auditability, regulatory compliance, and data retention policies.
12 | Continuous Improvement | Use operational metrics, incident reviews, and feedback loops to improve pipeline reliability and efficiency.
12 Core Principles of Good DataOps
DataOps (Data Operations) is a set of engineering practices that applies DevOps, Agile development, automation, and statistical process control to the management of data pipelines.
Its primary objective is to deliver reliable, high-quality, reproducible, and timely data to analysts, data scientists, machine learning systems, and business applications.
The 12 main principles
The 5 most important principles in practice
For a production data science or machine learning system, I would prioritize:
A particularly important principle: Statistical Process Control
One frequently overlooked aspect of DataOps is its relationship to statistical process control (SPC), borrowed from manufacturing.
Rather than merely checking whether a pipeline runs successfully, SPC helps determine whether the data-generating process remains statistically stable.
For example, in a financial ML pipeline, you might monitor:
A pipeline can execute perfectly while silently delivering statistically invalid data. Operational success does not imply statistical correctness.
great-wave commentedon Oct 27, 2026
Progress: I've updated the tutorial in google drive with a data validation exercise, and introduction paragraph about dataOps. I believe it is ready to be reviewed.
Blockers: This unused folder I created in the shared google drive can be deleted: https://drive.google.com/drive/folders/1pt-V9p-5T98LBZD1ILiESV6ghlJTlgq0?usp=drive_link
Availability: 6.
ETA: Pending review.
Pictures (if necessary): "Add any pictures that will help illustrate what you are working on."