Planning Under Uncertainty

This week we will read a pair of papers about POMDPs, Partially Observable Markov Decision Processes.  This framework has been used for robot planning and perception, as well as spoken dialogue systems.

• Leslie P. Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2), 1998.
• Hoey, Jesse, et al. Automated handwashing assistance for persons with dementia using video and a partially observable Markov decision process. Computer Vision and Image Understanding 114.5 (2010): 503-519.  Focus on section 3, the POMDP model.
  

Post by 5pm on Sunday about 25 words answering the following question:

• What are the challenges in using a POMDP model to drive a language-using agent?  How could that challenge be overcome? 

By 5pm on Monday, post a reply to someone else's blog post.  Suggest an alternate solution, ask a clarifying question, or point out something they might find useful.

Probabilistic Grounding Models (II)

This week we will continue our investigation into probabilistic grounding models.  We will read a paper about a language understanding system developed in collaboration between computational linguists and roboticists at the University of Washington.

• Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. A Joint Model of Language and Perception for Grounded Attribute Learning. In Proceedings of the International Conference on Machine Learning (ICML), 2012. 

This paper was written after the Chen and Mooney paper (2011) and the Tellex et al. paper.  (Even though the Tellex et al. paper hasn't come out yet, the original contributions appeared in conferences in 2010 and 2011.)  In some ways it unifies ideas from the two approaches:  it jointly learns a semantic parsing model as well as attribute classifiers for colors and shapes. 

Please post on the blog by 5pm on Wednesday a roughly 500 word answer to the following question:

• Compare and contrast how this paper represents word meanings with the previous week's readings.  (Chen and Mooney and Tellex et al.)  What is being learned?  What is given to the system?  What tradeoffs are being made by the this approach, compared to the other two?

Dialogue

This Tuesday's lecture will be a guest lecture by Scott AnderBois about dialogue.  The reading is here:

• Ginzburg, J. The Dynamics and Semantics of Dialogue. Logic, language and computation 1 (1996).

He won't be talking about the specific formalism, so you can mostly ignore the situation and infon-specific details.  He'll focus on the general mindset the paper embodies and about linguistic phenomena claimed to be sensitive to QUDs (Question Under Discussion).  There is no specific assignment for this week.  Instead I encourage you to begin working on your final project proposal.

UPDATE:  Scott's slides are here: http://cs.brown.edu/courses/csci2951-k/papers/QUD_Lecture_Robotics.pdf.

Probabilistic Grounding Models (I)

There were some questions in the comments about why we've been studying the readings so far, and also about how it relates to robotics.  As a result I've modified the schedule slightly, so that next week we jump into two papers about enabling robots to follow natural language commands.

The first paper describes an end-to-end system that my collaborators and I created for enabling robots to follow natural language commands about movement and manipulation.  The paper is currently in submission to a journal and not currently published, so please don't distribute it:

• Thomas Kollar, Stefanie Tellex, Matthew R. Walter, Albert Huang, Abraham Bachrach, Sachi Hemachandra, Emma Brunskill, Ashis Banerjee, Deb Roy, Seth Teller, Nicholas Roy.  Generalized Grounding Graphs: A Probabilistic Framework for Understanding Grounded Language.  (in submission.)   Read carefully through section 4, paying special attention to the math in Section 3.  Skim Section 5 (Route directions)

The second paper describes a different approach to following natural language commands by mapping between natural language and a symbolic language:

• David Chen and Raymond Mooney.  Learning to Interpret Natural Language Navigation Instructions from Observations.  AAAI 2011.  

Post a short (200 word) response to the following question: Compare and contrast how these two papers represent word meanings.  What is being learned?  What is given to the system?  What tradeoffs are being made by the two different approaches?

Tuesday, September 17: Grounded Semantics

This week we will read a paper from cognitive science describing the connection between spatial language and spatial cognition.  Specifically they study what geometric and perceptual features people appear to use to map between spatial language and objects in the external world. 

• Barbara Landau and Ray Jackendoff. “What” and “where” in spatial language and spatial cognition. Behavioral and Brain Sciences, 16:217–265, 1993.

For the assignment, let's try a different format to encourage more discussion.

• By Sunday night at 5pm, please post a comment on the blog of about 200 words about any of the things we have discussed so far.  You might describe a possible project, ask a question and present some possible answers, or compare and contrast ideas in what we've read so far and suggest areas you'd like to investigate more closely
• By Monday night at 5pm, please reply to at least one other comment.  Give them feedback about their ideas, try to answer their question, or expand on a point that you agree with.  I'm looking for about 200 words total; it could be spread across several different comments.

Thursday, September 12: Semantics

This week we will read about semantics and natural language.  My goal is for you to understand how words combine through syntax to create meaning.  In the following weeks we will study computational approaches, but this week we will focus on linguistic representations.

Read:

• I. Heim and A. Kratzer. Semantics in generative grammar, volume 13. Blackwell Oxford, 1998.  Chapter 1, 2.1, 2.2, 2.3, 2.5, 4.4.

Questions:

• Complete the  following mini-problem set:  http://cs.brown.edu/courses/csci2951-k/psets/2013-09-10-semantics/.  Download the latex file and fill in the entries in the table of word meanings using the notation from Heim and Kratzer.  Email me the tex and pdf file when you are done.  If you have never used latex before, there is more information about it at the Brown CS LaTeX page.
• Post an answer to the following question on the blog:  How should a robot represent word meanings?  What is good about the Heim and Kratzer approach?  What is missing?

Tuesday, September 10, 2013: Language Grounding and Artificial Intelligence

This Tuesday we will read one of the first systems for understanding natural language:

• Terry Winograd. Procedures as a Representation for Data in a Computer Program for Understanding Natural Language. PhD thesis, Massachusetts Institute of Technology, 1971. (pages 1-52)

Terry Winograd's thesis was a system for understanding language about a table top filled with blocks of different shapes and colors.  The whole thesis is more than 300 pages long; so only read the first two chapters.  Pay particular attention to Section 1.3, the sample dialog.   Please post a response of about 500 words¹ to the following questions in the forum:

• How is SHRDLU representing word meanings?  How is it combining word meanings together to understand entire sentences?
• What are the strengths and weaknesses of Winograd's approach for representing word meaning?  Especially think about running on a real robot instead of simulation.  For additional contex, skim Harnad's paper on the symbol grounding problem:  
• Stevan Harnad. The symbol grounding problem. Physica D, 43:335–346, 1990.
• What techniques and algorithms are necessary to support the types of linguistic behaviour that appear in Section 1.3?  Brainstorm!

1.  I'm looking for 500 words total, not 500 words for each question.

Welcome!

Welcome to Topics in Grounded Language for Robotics!  In the home, in the factory, and in the field, robots have been deployed for tasks such as vacuuming, assembling cars, and disarming explosives.  As robots become more powerful and more autonomous, it is crucial to develop ways for people to communicate with them.  Natural language is an intuitive and flexible way of enabling this type of communication.  The aim of this course is to study how to endow robots with the ability to interact with humans using natural language and then build systems!

The course will cover foundational material in artificial intelligence, computational linguistics, and robotics, as well as a survey of recent conference and journal papers.  A collaborative final project will provide an opportunity to more deeply engage with the material and provide a jumping-off point for future research.


We will use the course blog to post updates and host discussions about the papers.  Stay tuned for more details!