Data Engineer - 60009025

State of South Carolina - Department of Administration • Full-time • Richland, South Carolina, United States • $79.60k - $143.30k / year • 1w ago

Job Responsibilities

Data Engineer

The Department of Administration's Division of Enterprise Applications is seeking a Data Engineer to join our team. We are in growth mode, and it is an exciting time to be a contributing member to a team that maintains the SC Enterprise Information System (SCEIS)! SCEIS is the backbone of our business processes and ultimately empowers decision makers to improve government for the citizens. Under the reporting and analytics team, you will be responsible for designing, building, and maintaining scalable data Lakehouse solutions that support enterprise analytics, reporting, and data-driven decision making. This role focuses on developing robust data pipelines, interfaces for data access, automating data workflows, and ensuring high data quality and performance across the organization's data platform.

Responsibilities of the Data Engineer:

Design, build, and test ELT\ETL pipelines responsible for data ingress, curating, and warehousing of data based on functional and non-functional requirements.
Develop and maintain enterprise data cataloging processes to ensure datasets are properly documented, discoverable, and accessible, including managing metadata, data lineage, classifications, and integration with data governance tools.
Develop and manage the deployment of data pipelines and applications through CI\CD pipelines and\or GitOps based automation.
Work with architecture, product\service owners, and the development team to clarify work item functional and non-functional requirements including tasks needed to complete the work item so accurate effort estimates can be provided in backlog refinement meetings.
Evaluate new and emerging technologies to determine which ones can help innovate and\or implement new feature requests.
Participate in peer code reviews to ensure code quality and consistent use of patterns and styles.

Minimum and Additional Requirements

A bachelor's degree in computer science, information technology, or a related technical field and 2 years of experience in cloud-based data platforms (e.g. Azure Synapse, AWS Redshift, Google BigQuery, Databricks, Snowflake, etc.
Relevant experience may substitute for the bachelor’s degree on a year-for-year basis.

Additional Requirements:

Proven experience with data lakehouse architectures.
Expertise in designing and optimizing ELT\ETL workflows in distributed computing environments.
Proficient in writing modular, reusable, and testable code for data ingestion and transformation.
Python and SQL. Familiarity with Java is a plus.
Hands-on experience with Apache Spark.
Understanding of batch and real-time data processing paradigms.
Strong grasp of data modeling concepts (dimensional, normalized/denormalized, star/snowflake schemas) and data warehousing tools and concepts.
Experience working in agile development environments.

Applicants indicating college credit or degree(s) on the application will be required to bring a copy of college transcript to the interview. A copy of the transcript may also be uploaded as an attachment to the application, if required by the hiring department or if desired by the applicant. Please note that some areas of the Department may require an official, certified copy of the transcript prior to hiring or within a specific timeframe required by that area, after hiring. Failure to produce an official, certified transcript may result in not being hired or termination.

Additional Comments

The Department of Administration is committed to providing equal employment opportunities to all applicants and does not discriminate on the basis of race, color, religion, sex (including pregnancy, childbirth, or related medical conditions including, but not limited, to lactation), national origin, age (40 or older), disability or genetic information.

Supplemental questions are considered part of your official application. Any misrepresentation will result in your disqualification from employment. Please complete the state application to include all current and previous work history and education. A resume will not be accepted nor reviewed to determine if an applicant has met the qualifications for the position.

The South Carolina Department of Administration offers an exceptional benefits package for full time (FTE) employees:

Health, dental, vision, long-term disability, and life insurance for employees, spouse, and children. Click herefor additional information.
15 days annual (vacation) leave per year
15 days sick leave per year
13 paid holidays
Paid Parental Leave
S.C. Deferred Compensation Program available (S.C. Deferred Compensation)
Retirement benefit choices *
- State Retirement Plan (SCRS)
- State Optional Retirement Program (State ORP)

Enrollment in one of the listed plans is required for all FTE employees; please refer to the contribution section of hyperlinked retirement sites for the current contribution rate of gross pay.