پروژه پایتون 8031

---

شرح پروژه

سلام

من یه پروژه مربوط به دیتا ساینس داشتم

پروژه ام مربوط به data architecture and engineering هست

Project Description:

Go through the below scenario and present the design architecture. 

Also, take one component of the design, be it a container with the script, be it infrastructure as code, be it the API, be it the ELT/ETL script on a scheduler, you choose. We would like to see some code. 

Please find the task below:
Scenario

HS Health is looking for a new opportunity to expand on healthcare offerings. Recently in the news, the government announced their 10-year health plan to manage citizens' health, and provide better health care to citizens.

The Chief health officer from the government reached out to CEO of HS asking about program outcomes and asking if HS can help determine if current government initiatives are making an impact. A project team is formed with a data scientist and a data engineer from HS and an analytics team from the government. The role of the data scientist at HS is to evaluate the historical data and provide insights to the government. 

For the data scientist to be effective, the data must be readily available, and the dashboard is updated daily to start with, and going forward 4 times a day. This dashboard is what is made available to the government. 

Problem: 

The historical data is with the government and on their prem system, which cannot be consumed by data scientists. Since it's a transactional system it is difficult for the government to provide historical files daily. Hence the team at the government has made two APIs available to consume the daily and hourly data, one API which will provide daily data and one API providing hourly data. Parameters to the daily API is a date field and for the hourly data is a timestamp field. The government team has also provided historical files and has successfully landed in the S3 bucket on the HS side. Your job as a data engineer is to draft a high-level design so this data could be made available for the data scientist to consume from the warehouse. You also want to ensure that once dashboards are populated, are updated at least daily to start with, and in the next few months should be updated hourly. 

Note: 

1. Hourly is a wish list, this will delight the government and help them make decisions on an hour basis.
2. HS wants to keep infrastructure maintenance to a minimum.  

Task:

Prepare a presentation highlighting the high-level design of the solution. 
    - The visual diagram will be great to have. On the day we may have questions but not limited to

•What tools you are aware of and what ones you have no to little experience with, will you use to design or craft your solution? 

•How will you ensure given the sensitivity of data, your design will adhere to security and least privileges access? 

•How will you justify the running and managing costs of your design? 

•Any tips or standards you want a data science person to follow, so the solution is clean, coding practices are adhered to (based on your previous experience, assumptions), etc? 

Format of presentation:

•Powerpoint or GitHub readme MD or any other tool of your choice. 

•Feel free to add some code examples in the slides, dirty code is welcome, but please add comments if you use code to explain.

Task duration (1-2 hours). We believe 2 hours should be sufficient, but if you want go beyond that, that is up to you. We don't want super pretty presentations; just simple ones that could explain why you chose the solution you chose. (2-4pages if using PPT)

مهارت ها و تخصص های مورد نیاز

---

---