Name: News and Advice on the World's Latest Innovations | ZDNET
Brand: zdnet.com
SKU: zdnet-com-460
Availability: OnlineOnly

critic

News and Advice on the World's Latest Innovations | ZDNET

preview

/&gt;The time has come, said Re, for AI scientists to focus on higher-level tasks such as bringing domain expertise to AI rather than twiddling hyper-parameter settings of neural networks.Some AI researchers' practices are as tired as a Michael Bay movie, to hear Christopher Re tell it.&nbsp; Wednesday, Re, who is a Stanford University associate professor of computer science, for the University's Human-Centered Artificial Intelligence institute. His topic: "Weird new things are happening in software." That weird new thing, in Re's view, is that the stuff that was important only a few years ago is now rather trivial, while new challenges are cropping up. The obsession with models, meaning, the particular neural network architectures that define the form of a program, has run its course, said Re. Re recalled how in 2017, "models ruled the world," with the prime example being Google's Transformer, "a much more important Transformer than the Michael Bay movie that year," quipped Re. But after several years of building on Transformer -- including Google's BERT and OpenAI's GPT -- "models have become commodities," declared Re. "One can pip install all models," just grab stuff off the shelf. What Re termed "new model-itis," the obsession by researchers to tweak every last nuance of architectures, is just one of many "non-jobs for engineers" that Re disparaged as something of a waste of time. Tweaking the hyper-parameters of models is another time waster, he said.&nbsp; Instead, Re told the audience, for most people working in machine learning, "innovating in models is kind-of not where they're spending their time, even in very large companies," he said.&nbsp; "They're spending their time on something which is important for them, but is also, I think, really interesting for AI, and interesting for the reasoning aspects of AI." Where people are really spending time in a valuable way, Re contended, is on the so-called long tail of distributions, the fine details that confound even the large, powerful models. "You've seen these mega-models that are so awesome, and do so many impressive things," he said of Transformer. "If you boil the Web and see something a hundred times, you should be able to recognize something." "But where these models still fall down, and also where I think the most interesting work is going on, is in what I call the tail, the fine-grained work." The battleground, as Re put it, "are the subtle interactions, subtle disambiguations of terms," what Re proposed could be called "fine-grained reasoning and quality." That change in emphasis is a change in software broadly speaking, said Re, and he cited Tesla AI scientist Andrej Karpathy, who In fact, Re's talk was titled "machine learning is changing software."&nbsp; Re speaks with real-world authority over and above his academic legacy. He is a four-time startup entrepreneur, having sold two companies to Apple, Lattice, and Inductiv, and having co-founded one of the many fascinating AI computer companies, . He is also a MacArthur Foundation Fellowship recipient. (.) To handle the subtleties of which he spoke, Software 2.0, Re suggested, is laying out a path to turn AI into an engineering discipline, as he put it, one where there is a new systems approach, different from how software systems were built before, and an attention to new "failure modes" of AI, different from how software traditionally fails.&nbsp; Also:&nbsp;It is a discipline, ultimately, he said, where engineers spend their time on more valuable things than tweaking hyper-parameters. Re's practical example was a system he built while he was at Apple, called Overton. Overton allows one to specify forms of data records and the tasks to be performed on them, such as search, at a high level, in a declarative fashion. Overton, as Re described it, is kind-of an end-to-end workflow for deep learning. It preps data, it picks a model of neural net, tweaks its parameters, and deploys the program. Engineers spend their time "monitoring the quality and improving supervision," said Re, the emphasis being on "human understanding" rather than data structures.&nbsp; Overton, and another system, , developed by Uber machine learning scientist Piero Molino, are examples of what can be called zero-code deep learning.&nbsp; "The key is what's not required here," Re said. "There's no mention of a model, there's no mention of parameters, there's no mention of traditional code." The Software 2.0 approach to AI has been used in real settings, noted Re. Overton has helped Apple's Siri assistant; the built by Re and collaborator Stephen Bach contributes to Google's advertising technology. And in fact, has been turned into run by lead Snorkel developer Alexander Ratner, who was Re's graduate student at Stanford. "Lots of companies are using them," said Re of Snorkel. "They're off and running." As a result of the spread of Software 2.0, "Some machine learning teams actually have no engineers writing in those lower-level frameworks like TensorFlow and Pytorch," observed Re. "That transition from being lab ideas and weirdness to actually something you can use has been staggering to me in really just the last three or four years." Re mentioned other research projects at the forefront of understanding the tail problem. One is , developed by Re and Simran Arora and others at Stanford, which makes advances in what is called named entity disambiguation. For questions such as "How tall is Lincoln," knowing that "Lincoln" means the 16th U.S. president, versus the car brand, is one of the those long tail problems. Also: Another research example of more high-level understanding was a system Re with Stanford researchers Nimit Sohoni and colleagues called George. AI-based classifiers often miss what are called subclasses, phenomena that are important for classification but are not labeled in training data.&nbsp; The George system uses a technique called dimensionality reduction to tease out hidden subclasses, and then train a new neural network with that knowledge of the subclasses. The work, said Re, has great applicability in practical applications such as medical diagnosis, where the classification of disease can be mislead by missing the subclasses.&nbsp; Work such as George is only an early example of what can be built, said Re. There is "lots more to do!" The practice of Software 2.0 offers to put more human participation back in the loop, so to speak, for AI.&nbsp; "It's about humans at the center, it's about those unnecessary barriers, where people have domain expertise but have difficulty teaching the machine about it," Re said.&nbsp; "We want to remove all the barriers, to make it as easy as possible to focus on their unique creativity, and automate everything that can be automated."

/&gt;The time has come, said Re, for AI scientists to focus on higher-level tasks such as bringing domain expertise to AI rather than twiddling hyper-parameter settings of neural networks.Some AI researchers' practices are as tired as a Michael Bay movie, to hear Christopher Re tell it.&nbsp; Wednesday, Re, who is a Stanford University associate professor of computer science, for the University's Human-Centered Artificial Intelligence institute. His topic: "Weird new things are happening in software." That weird new thing, in Re's view, is that the stuff that was important only a few years...

public

Tiernan Ray

ZDNet is a business technology news website published by CBS Interactive, along with TechRepublic. The brand was founded on April 1, 1991, as a general interest technology portal from Ziff Davis and evolved into an enterprise IT-focused online publication owned by CNET Networks.

News and Advice on the World's Latest Innovations | ZDNET

No article scores yet.

CRITIC REVIEWS

PUBLIC REVIEWS