There has been lots of interest in gpt-2 recently - it's taught itself to "talk" and to play chess. If you've not heard about it, it's a neural network that has learned to produce language using a massive database of language samples scraped from the internet and a huge number of tune-able parameters, but without any explicit knowledge about the rules of language.
As Gary Marcus points out in this excellent and accessible article, gpt-2 offers a new testing ground for theories of language acquisition and whether humans come pre-wired with tools for learning language or whether language can be learned from the input statistics alone. I've just begun messing around with the model using the Python package: gpt2-simple. It's nice and easy to use, the only snag was having to depreciate TensorFlow to an older version to make it run. I used the "small model" - trained on 8 million web pages - referred to as 124M and fine-tuned the model "for fun" with the conference program of the Society for Neurobiology of Language Conference 2019. I ran 1000 training iterations without using GPU and it finished in around 5 hours on an Ubuntu system: Intel® Core™ i5-8400 CPU @ 2.80GHz × 6 (8 GB RAM). So here is what gpt-2 thinks that language science is all about ... The all important welcome information, after all, every conference program needs a strong opener: "Welcome to the Online SNL 2019 Program! SNL 2019 is upon us! SNL 2019 brings together the leading researchers from different disciplines in a game-changer for neurolinguistics in neurotypicalism. This includes a whole new venue for neurolinguistics in neurotypicalism, a whole new venue for neurolinguistics in neurotypical behavior, and a whole new venue for neurolinguistics in the brain. Participants from around the globe travel to South America, the Caribbean Sea, and the Indian Ocean to attend the Opening Night Reception, which features some of the best and the worst from across the neurolinguistics fields." On a more serious note, gpt-2 does a decent job with poster titles: D84 Lingering Pronunciation Constraints during Naturalistic Intervals of Word Form B76 Hemispheric parallel access for orthographic comprehension B77 Interactions of orthographic depth and semantic complexity in proficient spoken and written language Comprehension. B3 Prediction: Agents and Predictors of Reading B33 Sparse canonical correlation analysis of reading: when and how? A38 Steady state analysis of the fMRI fMRI ERP lexicon E22 A new perspective on pathology in post-stroke aphasia: Investigating the processes underlying recovery from stroke and improving surgical outcome is an essential first step in translational therapy trials B53 Rule learning in reading: Evidence from ERPs and PET E85 Distinct neural substrates of visual and auditory processing converged in temporoparietal cortex during L2 activation And also pretty good on full abstracts (although note, as Gary Marcus' article points out not every thing that comes out is sensible, I'm cherry picking the better ones): "A key question in the neuroscience of language is whether masses of individuals engineer their own thought processes via algorithmic processes or they form independent trials driven by their own algorithmic decisions. Theories of Brain and Mind are diverse; some posit that a core function of language production is in the making of conceptual sentences; while other views propose that language production is a collection of experiential functions, that it must choose its objects, events or minds and therefore must choose between semantic and introspective features. The present study is to investigate whether a core function of language production is composing conceptual sentences in a non-attentional capacity. We used a fast-event inference (FIF-ICA) paradigm to investigate the neural bases of this challenging behaviour. We used a fast-parse approach to investigate the neural bases of this challenging behaviour. We studied 150 healthy young volunteers for about five minutes whilst viewing pictures of familiar objects. The participants had difficulty forming coherent sentences as a result of aneuplasia of the premotor cortex (PMC). We observed two patterns of activation in our analyses of these images a few hours after the face-off. A central region of interest (COWI) was positively disrupting when object naming produced differences between the two groups of healthy volunteers. Diffusion-tensor imaging (DTI) data were acquired for all participants at a resolution of 1.5mm by 2mm. We focused on regions of interest (ENC) which direct visual and auditory cortices, implicated in language production. We observed a relatively widespread activation pattern across the cortical hemispheres, with disproportionately distributed areas of the bilateral hemisphere in healthy individuals. While there was limited evidence in the previous a few years that these areas are sensitive to cognitive changes, we now showed that these encodings are gradually becoming more detectively correlated over time. We found that the bilateral lateral cerebellum, which is involved in visual word production, was also being observed to be affected by these changes. We discussed below the neuroimaging results with the aim of identifying regions of interest in the context of picture-naming that might be related to these neuro-treasuring processes. The regions of interest were identified using fBOI, which is a resting-state functional-functional magnetic resonance imaging (rfMR) computerized imaging tool. More specifically, we investigated the effect of changes in resting-state functional connectivity (rsFCC) on picture naming in healthy ageing. We found that over the presence of aged objects, rsFCC became more dependent on predictive coding than on any other domain in the domain. We also found that rsFCC evoked a greater response of the WABC than any other domain in the >4-year-old group. Thus, a core function of language production is being spatially and spectrally correlated with the development of language production. Future studies on this subject will be emphasized at both the state and domain level." "The neurobiology of the phonological ratio, between Heschl’s and Deaf listeners, is a debated topic in neurobiology. In the current study, we used magnetoencephalography (MEG) to understand the timing and correlates of the acquisition of phonological sequences. Twenty-four participants (age 16-49, range 23-64) were tested in a visual novel Word-to-Word trial with three conditions: (1) an articulatory target word with a lexical layout phonologically associated with the letter / / (versus /sh//), (2) a non-agricultural target word with a lexical layout phonologically related to the letter /i/ (versus /s/), and (3) a semantic target word with a phonological revision based on the letter /i/ (versus /sh/). The patterns of detection were further used to be refined to include the components of phonological integration, termed “phonological artifacts”. These components, known as “non-agricultural artifacts”, were acquired as free samples, and were quantified using MEG. Non-agricultural artifacts were identical in both hemispheres. Non-European languages (e.g., Finnish) were used as controls. Both parental language (English) and her native language (Norwegian) were used as controls. The brain activity of the non-agricultural artifacts was analyzed by calculating the inter-hemispheric disconnection (IHDC) between the cortical areas supporting the phonological processing of the pitch and frequency domain. We found that the cortical circuits supporting phonological word retrieval were disconnection induced by the presence of the non-agricultural artifacts in the same regions as the cortical areas supporting word identification, namely, the basal, middle, and inferior temporal language areas. In addition, we observed that the cortical circuits supporting syntactic processing, namely, the dorsolateral and ventral language areas, were disconnection induced by the presence of the non-agricultural artifacts in the same regions as the cortical circuits supporting phonological processing, namely, the insula and basal language areas. Direct cortical mapping of the inferior and middle temporal areas is proposed to support the development of the reading network." So that's decided, next year gpt-2 is writing my abstracts whilst I put my feet up...I'm also happy to lend my trained model to the SNL organising committee, for a small fee.
1 Comment
|
Samuel EvansCognitive Neuroscientist and Lecturer Archives
January 2020
Categories |