Author: Sheri Wells-Jensen.
Dr. Wells-Jensen is an Associate Professor and Co-Director of the ESOL Program at Bowling Green State University. She also coordinates BGSU's Minor in Linguistics. Her teaching and research interests include phonetics, applied phonology, psycholinguistics, speech production (especially slips of the tongue), language preservation, braille and xenolinguistics.
Here comes the zillion-dollar question:
If humans encounter one, would we be able to learn an alien language?
OK, because I can't actually answer that question, I'm going to try to distract you with some other clarifying questions and link to a bunch of additional reading material so that, by the time you get to the end of this, you won't feel too bad that I had nothing definitive to say.
What do you mean by “encounter”?
The best way to learn a language is face to … um … “face”: The intrepid linguist ventures boldly into the alien lair and goes all Amy-Adams at them, gesturing, pointing to things, writing key words on a white board. But first contact is much more likely to be via an enigmatic radio signal, rather than large, exciting spaceships in orbit. Although both are excellent reads, it’ll be more like Carl Sagan’s Contact (https://www.goodreads.com/book/show/61666.Contact) than Arrival or its source material, Ted Chiang’s novella “Story of Your Life”. (https://www.goodreads.com/book/show/223380.Stories_of_Your_Life_and_Others) Still … if it does happen, send the government helicopter to collect me; I'm ready to go! (Seriously.)
What do you mean by “encounter”? (Part 2)
Do you mean you think it will be a spoken language? We have to consider the medium, that is, how the language is conveyed. If it is "spoken", it could be wildly fast, brain-achingly slow, above our range of hearing, below our range of hearing, or made of details too subtle for us to hear. Or it could be not-spoken—conveyed by visible physical movement (like the hundreds of currently-known human sign languages), color changes, emitted infrared light or magnetic pulses, or through direct tactile stimulation. Read Hellspark by Janet Kagan (https://www.goodreads.com/book/show/1056201.Hellspark) for some good fun on this point.
What do you mean by “we”?
It's really little kids that are good at language learning, not middle-aged linguists. Toddlers seem to be wired to learn languages, while adults muddle by and do less well. Anyone up for an interplanetary education exchange involving … our 2-year-olds? Not me, thanks. The Sparrow by Mary Doria Russell (https://www.goodreads.com/book/show/334176.The_Sparrow) mentions this kind of arrangement.
What do you mean by “learn”?
My cat knows the word “supper”. Most dogs know the word “walk”. Some dogs, parrots, and chimpanzees know hundreds of individual words, but they can't make sentences. That might be all we could manage. My favorite (nonfiction) peek into an alien mind trying to communicate in words is Alex and Me by Irene Pepperberg. (https://www.goodreads.com/book/show/3018307-alex-me)
What do you mean by “learn”? (Part 2)
We should also consider the infuriating possibility that they could learn our languages but we could not learn theirs. There’s no real reason why language learning would have to be reciprocal. There's a cool depiction of an alien linguist in Robert Heinlein's classic The Star Beast. (https://www.goodreads.com/book/show/175328.The_Star_Beast)
What do you mean by “alien”?
There's some evidence that the way our bodies are built (standing erect with two hands to manipulate objects and our particular standard set of sensing organs) has a lot to do with what kind of language we speak. If that's true, we might be able to manage the language of aliens who are roughly humanoid, but the language spoken by sentient gas bags or intelligent snails would be forever beyond us. Expose a human child to a non-humanoid language and poof!—Her little brain explodes. This is interestingly depicted (though without the gory details because she's nice like that!) in Suzette Haden Elgin's Native Tongue. (https://www.goodreads.com/book/show/285563.Native_Tongue)
So, the cumulative answer to the original question—“would we be able to learn an alien language?”—is “Probably not / Maybe / I don't even want to / I think so / Who knows? / Some of them, probably.”
And that's about the best we can do for now.