This is a solution for a ExtBase Backend Extension with Ajax calls (usually required for ExtJS). It is based on this solution, which required lots of debugging until it worked for me (on 4.5): http://daniel.lienert.cc/blog/blog-post/2011/04/23/typo3-extbase-und-ajax/

This is the Ajax Request – notice „actionName“ instead of „action“ and lowercase/uppercase spellings:

Ext.Ajax.request({
  url:'ajax.php',
  params: {
    ajaxID: 'importexcelAjaxDispatcher',
    request: Ext.encode({
      extensionName:'ImportExcel',
      pluginName:'importexcel',  //this is from ext_tables.php, Tx_Extbase_Utility_Extension::registerModule(x,x,pluginName,...), often Pi1 or mod1
      controllerName:'ImportExcel',
      actionName:'testAjax',
    })
  },
  success:function(response, request) {
    //do something
  },
  failure: function(response, request) {
       console.log('server-side failure with status code ' + response.status);
  }
});

Then register the Ajax Dispatcher in ext_localconf.php:

$TYPO3_CONF_VARS['BE']['AJAX']['importexcelAjaxDispatcher'] = t3lib_extMgm::extPath($_EXTKEY).'Classes/Domain/Utility/AjaxDispatcher.php:Tx_ImportExcel_Domain_Utility_AjaxDispatcher->initAndDispatch';

Get the Ajax Dispatcher from pt_extbase/Classes/Utility/AjaxDispatcher. For any unknown reason, the response is still a full HTML page, so here is my modification of dispatch function. I check the successfull calls with devLog.

public function dispatch() {
        t3lib_div::devLog('Ajax dispatch','import_excel',0,array('ExtensionName'=>$this->extensionName,'PluginName'=>$this->pluginName));

        $configuration['extensionName'] = $this->extensionName;
        $configuration['pluginName'] = $this->pluginName;

        $bootstrap = t3lib_div::makeInstance('Tx_Extbase_Core_Bootstrap');
        $bootstrap->initialize($configuration);

        $this->objectManager = t3lib_div::makeInstance('Tx_Extbase_Object_ObjectManager');

        $request = $this->buildRequest();
        $response = $this->objectManager->create('Tx_Extbase_MVC_Web_Response');

        $dispatcher =  $this->objectManager->get('Tx_Extbase_MVC_Dispatcher');
        $dispatcher->dispatch($request, $response);       

        //parse content from HTML
        preg_match('/!#!#!(.*)!#!#!/',$response->getContent(),$match);
        t3lib_div::devLog('Ajax response','import_excel',0,array($content,$match));

        $response->sendHeaders();
        if ($match[1]) {
            echo $match[1];    
        } else{
            echo $response->getContent();
        }

        return;
    }

And finally my Action:

class Tx_ImportExcel_Controller_ImportExcelController extends Tx_Extbase_MVC_Controller_ActionController {

public function testAjaxAction() {
        t3lib_div::devLog('testAjaxAction called','import_excel',0,array());
         $test = array('test'=>1, 'test2'=>'Hello from Extbase');
         $result = json_encode($test);
         return '!#!#!'.$result.'!#!#!';
    }
}

View „TestAjax.html“ is an empty file.