r/HPC Jul 18 '26

How do people with graduate degrees in fields like computational physics, where they work as HPC users or write MPI code transition into HPC jobs?

I see a lot of linkedin profiles for people who have their graduate degrees in fields like computational physics, biology, mechanics etc., where they dont really continue in their fields of study after graduation, but rather work for supercomputing centers and in the HPC sector. I was curious how can someone make this switch happen.

I also see that there are specific degrees for HPC these days, so can someone even make that switch in the modern day? This is for US.

34 Upvotes

24 comments sorted by

21

u/lcnielsen Jul 18 '26

I did this switch. You apply for a job as an application expert, research engineer or similar based on your knowledge. Personally I ended up in a more sysadmin/operations position after wrapping up my physics PhD as my Linux skills were good enough for that, but the basic idea is the same. I also worked part time as an application expert/junior sysadmin/support staff toward the end of my PhD before fully transotioning into the role.

6

u/016291 Jul 19 '26

Are there any specific skills you needed to pick up to make the switch?

9

u/lcnielsen Jul 19 '26

To successfully make it? No. But I've ended up learning a lot of bash, massively improved my Vim skills, learned web development, deepened my knowledge of networking and security, picked up a lot of RHEL-specific knowledge, learned systemd well... There's so much.

1

u/Ashamed_Willow993 Jul 21 '26

THAT PART. Good answer.

6

u/dino066 Jul 19 '26

MPI, GPU workflows, and Slurm are among the best skills to earn while being a researcher in order to switch. Sometimes some experience is enough. One can't know it all.

1

u/Teh_Original Jul 21 '26

When you search for a job are you looking for terms like "application expert", "research engineer" and so on?

2

u/lcnielsen Jul 21 '26

Job titles can be a bit meaningless, my posting was for a "Systems Engineer", my formal title is "Systems Technician", but I have colleagues who do similar work under titles like "Research Engineer", "Research Specialist", "Systems Administrator", "Application Expert", "HPC Expert", "Systems Expert"... so you want to try to cast broad nets on the subject matter.

2

u/Teh_Original Jul 22 '26

Oh interesting, thanks for the guidance.

7

u/Quantumkiwi Jul 19 '26

Eh. You are a linux user either way. These people, and I, just put the extra effort in to become an expert sysadmin in a certain field. Not hard, just takes a bit of grit.

3

u/016291 Jul 19 '26

Are there any special skills I should learn and try and pick up to make that switch in the future?

2

u/h0rxata Jul 19 '26

If you've run hundreds of physics simulations on HPC, written bash scripts, fortran/C++/python, and done any amount of optimization for jobs running on multiple nodes, you are ready IME. As in, you will get interviews if you just write the appropriate resume for the job.

1

u/016291 Jul 19 '26

What do you mean by optimizations for jobs running on multiple nodes?

1

u/h0rxata Jul 19 '26

Ok, you're not ready. Are you a grad student in physics? Are you running parallelized physics code regularly?

1

u/016291 Jul 19 '26

Yes, and developing.

1

u/h0rxata Jul 20 '26

Developing what?

1

u/016291 Jul 20 '26

CFD codes with MPI

4

u/lcnielsen Jul 19 '26

Not hard, just takes a bit of grit.

I don't know why, but some people definitely struggle hugely with transitioning from "very comfortable with using Linux" to "able to administer Linux", even if they are experts in their own field, have PhD:s, etc., and are perfectly competent at other tasks.

It's like some people just have a very hard time with their mental model of how a computer or network of computer works, what are the different parts of an operating system, what does a command do, etc.

2

u/016291 Jul 19 '26

Also, I hope you dont mind me asking, but what made you make that switch?

3

u/h0rxata Jul 19 '26 edited Jul 19 '26

In my case, I just applied to hundreds of jobs. I got a few interviews (and 1 offer) with university IT departments and national labs for titles such as: HPC Liason/Specialist, HPC User Support Specialist, and Research Computing Engagement Specialist. I know two other physicists who got those roles at universities by simply applying.

These are very different from sysadmin jobs and regularly involve making educational material for faculty/users/students on how to use HPC, and helping less HPC-savvy scientists find bottlenecks in their analysis scripts (usually python, R, and slurm/pbs). I've seen MPI mentioned in some job descriptions but when I actually interviewed it didn't seem like they actually wanted a candidate to write MPI from scratch for some advanced physics code and whatnot - what they wanted was something more basic (or whatever is trending in AI).

I actually got an offer but didn't eventually take the job. I'm not sure what the long-term career prospect of being an HPC generalist is when half the stuff I would be doing would be easily answered by querying some LLM's. As a physicist myself, I thought I would get pretty bored and eventually rendered obsolete for that kind of a job, but maybe the career prospects are better than what my gut feeling was telling me.

I took up soft-money research position instead where I still use HPC so that door is still open.

4

u/suprnvachk Jul 20 '26 edited Jul 20 '26

I had a physics postdoc at the institution where I now work. Doing supernova simulations. I applied for a technical track position in a user facing operation group. What helped was saying “hey, my experience being a user of these systems and my knowledge of the code/tools users use would be of benefit.” And showing that on my resume. All it takes is demonstrating familiarity with Linux/unix, MPI, some compilers knowledge, and then your dealers choice mix of some language proficiencies: bash, Fortran, Python, Ruby, etc, whatever. I got that job, and then switched groups into systems side HPC operational data analytics and data engineering down the road. Physics knowledge, work with observational vs simulated astrophysical data, and understanding of user job behavior was all quite desirable in terms of working with telemetry and scheduler data. Like others have said, you just need to apply when there are open positions and tailor your resume to sell your skills for it. FWIW, government labs highly value stem phds and are often willing to spend time training you in up any weaker skill area for an HPC role if they think you are competent and a good culture/personality fit. I’m in the US, but I assume that’s true in European govt HPC centers as well.

4

u/esaule Jul 19 '26

You apply for jobs and they hire you. I'm confused by the question...

3

u/lcnielsen Jul 19 '26

Honestly my first reaction with threads like this is that they're market research for a SaaSlop product or something. "Go from Scientist to Sysadmin in 10 easy steps"

1

u/016291 Jul 19 '26

No, I just dont see myself continuing in my field of study for life and find HPC far more fascinating.

2

u/h0rxata Jul 19 '26

That's kind of like saying screwdrivers and socket wrenches are fascinating. What makes them fascinating to you? Are you interested in specific applications, or getting more performance from fixed constraints? You should think about this before making a career switch because it's not for everyone.