I searched around quite a bit and found a lot of incomplete walkthroughs and information on setting up claims-based authentication for Sharepoint 2010. Here is the process I finally ended up using and some extra stuff that may be helpful. This seems to be the most direct route to getting things set up, as you don’t have to fiddle around in IIS, just edit your xml config files.
Just a notice, I am using SharePoint Server Enterprise 2010 with SQL Server R2 Standard on Windows Server 2008 R2 Standard, but your SQL can be anything, even express and SharePoint can be just WSS/SharePoint Foundation 2010.
Fist you need to open Visual Studio and create a new web site. This didn’t make sense to me at first, either, but it is only to create your SQL database for .NET users. The aspnet_regsql command did not work for me, so I used VS2010 instead(any version that uses .NET 2.0 should be fine as that is Sharepoint 2010′s default .NET settings). So for your new site, just pick a blank C# site, make sure it is .NET 2.0.

Creating new website in Visual Studio 2010
Then you will go up top to the “Website” menu and select “ASP.NET Configuration”. This will open a site that we can configure .NET authentication database. We will go to the Providers tab first. Choose the option “Select a different provider for each feature(Advanced)”. Test the connection for both AspNetSqlMembershipProvider and AspNetSqlRoleProvider.
Next, go to Security tab and select authentication type under users. Pick “From the internet”, then click done. At this point, I would create a test user and a couple roles in the database (I just started with “standard” and “superuser” for the test roles).
Browse to the location where you created the web site (i.e. C:\Users\<user>\documents\Visual Studio 2010\web sites\<website1>\App_Data) and copy the ASPNETDB.mdf and the ASPNETDB.ldf (sometimes ASPNETDB_log.ldf or other variant) to your SQL server directory. Default directory for SQL 2008 R2 on Server 2008 R2 is: C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA.
Now close Visual Studio and delete the directory for that web site. We just needed those database files to be created.
Once the files are in the appropriate directory(this may be a seperate directory, if you do not keep your db files in the default location), we will open SQL Server Management Studio and attach the database. To attache the database, right-click on “Databases” and select “Attach”.
Then, from the next screen you click “Add…” and select the ASPNETDB.mdf file. It will then populate the other fields like so:

One thing that is important is to verify the path on the bottom is correct for the “current file location” for your log file (.ldf).
Now we will create our application & site collection in Sharepoint. Open Central Admin, click “Manage Web Applications” under the Application Management heading. Then select “New” from the ribbon up top. In the Create New Web Application box, there are only 2 sections that you will need to modify to get claims-based auth to work, the rest you can set however you see fit for your environment. First, make sure that Claims Based Authentication is selected as the Authentication method. Second, you sill need to set the providers for your forms based authentication. To do this, next to the Claims Authentication Types setting, leave The Windows authentication as it is, but check “Enable Forms Based Authentication (FBA)” and fill in the boxes as seen below:

Now, create your site collection by going to Application Management in SharePoint Central Admin console. Then Select Create site collections under the Site Collections heading. Select the application we just created, give it a title and whatnot and pick your site collection admins. I like to create the site now, because then it forces you to choose admins who can access via Windows auth, in case the FBA config doesn’t work out.
And you are pretty much ready to go, except for editing the xml configuration files. Now, before we proceed, make a backup of these xml files. This is very important. The first file we will be modifying is the Central Admin web.config. It is located at: C:\inetpub\wwwroot\wss\VirtualDirectories\<port>\web.config by default.
Between </appSettings> and </configuration>, insert the following. If there is another section named <connectionStrings>, delete it. Please do not comment anything out in here, unless you know XML well. Comments in the wrong places can cause XML to become unreadable.
<connectionStrings>
<add name=”SQLConnectionString” connectionString=”data source=<SQL Server>;Integrated Security=SSPI;Initial Catalog=ASPNETDB” />
</connectionStrings>
Next, right before the close of system.web or </system.web>, insert the following. Again delete any duplicates of <membership> or <roleManager> categories and delete everything ocntained within them as well (i.e. all between <membership> and </membership>).
<roleManager defaultProvider=”AspNetWindowsTokenRoleProvider” enabled=”true” cacheRolesInCookie=”false”>
<providers>
<add connectionStringName=”SQLConnectionString” applicationName=”/” name=”SQLRoleManager” type=”System.Web.Security.SqlRoleProvider, System.Web, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” />
</providers>
</roleManager>
<membership defaultProvider=”SQLMembershipProvider”>
<providers>
<add connectionStringName=”SQLConnectionString” passwordAttemptWindow=”5″ enablePasswordRetrieval=”false” enablePasswordReset=”false” requiresQuestionAndAnswer=”true” applicationName=”/” requiresUniqueEmail=”true” passwordFormat=”Hashed” name=”SQLMembershipProvider” type=”System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” />
</providers>
</membership>
Save that file and we’re on to the next. The web application’s web.config, which will be located by default at: C:\inetpub\wwwroot\wss\VirtualDirectories\<port>\web.config. Again, make a copy before editing. In between the </SharePoint> and </system.web>, insert the following (the same “delete duplicates applies to all xml changes, so I’m going to stop mentioning it).
<connectionStrings>
<add name=”SQLConnectionString” connectionString=”data source=<SQL Server>;Integrated Security=SSPI;Initial Catalog=ASPNETDB” />
</connectionStrings>
Now, in between your <machineKey …> and the </system.web> insert the following.
<roleManager enabled=”true” defaultProvider=”c”>
<providers>
<add name=”c” type=”Microsoft.SharePoint.Administration.Claims.SPClaimsAuthRoleProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c” />
<add connectionStringName=”SQLConnectionString” applicationName=”/” name=”SQLRoleManager” type=”System.Web.Security.SqlRoleProvider, System.Web, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” />
</providers>
</roleManager>
<membership defaultProvider=”i”>
<providers>
<add name=”i” type=”Microsoft.SharePoint.Administration.Claims.SPClaimsAuthMembershipProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c” />
<add connectionStringName=”SQLConnectionString” passwordAttemptWindow=”5″ enablePasswordRetrieval=”false” enablePasswordReset=”false” requiresQuestionAndAnswer=”true” applicationName=”/” requiresUniqueEmail=”true” passwordFormat=”Hashed” name=”SQLMembershipProvider” type=”System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” />
</providers>
</membership>
Alright, save it and we’re on to the Security Token Serivce’s configuration. This one is located at: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\WebServices\SecurityToken\web.config by default. Here, we are going to insert this right in between your </system.net> and </configuration>, but if you already have a <system.web> category in this section, please be sure to copy your <machineKey… > entry from the old system.web and put it into this new one (then delete the old).
<system.web>
<roleManager defaultProvider=”c” enabled=”true” cacheRolesInCookie=”false”>
<providers>
<add name=”c” type=”Microsoft.SharePoint.Administration.Claims.SPClaimsAuthRoleProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c” />
<add connectionStringName=”SQLConnectionString” applicationName=”/” description=”Stores and retrieves roles from SQL Server” name=”SQLRoleManager” type=”System.Web.Security.SqlRoleProvider, System.Web, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” />
</providers>
</roleManager>
<membership defaultProvider=”i”>
<providers>
<add name=”i” type=”Microsoft.SharePoint.Administration.Claims.SPClaimsAuthMembershipProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c” />
<add connectionStringName=”SQLConnectionString” passwordAttemptWindow=”5″ enablePasswordRetrieval=”false” enablePasswordReset=”false” requiresQuestionAndAnswer=”true” applicationName=”/” requiresUniqueEmail=”true” passwordFormat=”Hashed” description=”Stores and Retrieves membership data from SQL Server” name=”SQLMembershipProvider” type=”System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” />
</providers>
</membership>
</system.web>
<connectionStrings>
<add name=”SQLConnectionString” connectionString=”data source=<SQL Server>;Integrated Security=SSPI;Initial Catalog=ASPNETDB” />
</connectionStrings>
Now, you can save this file and go to your site you created earlier. You will log in with Windows authentication, this time and then in site permissions, add the test user you created in Visual Studio earlier to the site with whatever permissions you like. Or, add the two roles (standard & superuser) with the rights you want them to have. i.e. add the “standard” role from SQL to the site members group in Sharepoint. This will all depend on your group and permissions structure.
An important note, the people picker normally can search partial names to find users in AD, but for FBA, it cannot do the search for user and group names. If your user/role is named “test1″, you cannot enter test and let it check namees to search for it. You need to type the full name in, then click check names and it should show that the name either came from active directory or your FBA database. Below, I show how you can tell whether it sees the users from FBA DB. I have two accounts with the same username in SQL and in AD. people p[icker will show you which it is like so:

Now, sign out of the sharepoint site and try to sign in using the user account from forms auth db. Should be good to go.
So, now your site is working for claims-based authentication, but there are some other disclaimers about things I ran into along the way. Hopefully, these extra little nuggets save you some time.
First, you may receive an error like so when using FBA or resetting passwords:
You must specify a non-autogenerated machine key to store passwords in the encrypted format. Either specify a different passwordFormat, or change the machineKey configuration to use a non-autogenerated decryption key.
See this MSDN article for information on Machine Keys. Honestly, you will most likely just want a key that you can use for your site. This auto-generater may help.
Also, you will see that when you try to access .NET Users or .NET Roles via IIS manager, you will get this error and not be able to manage users:
This feature cannot be used because the default provider type could not be determined to check whether it is a trusted provider.
You can use this feature only when the default provider is a trusted provider. If you are a server administrator, you can make a provider a trusted provider by adding the provider type to the trusted providers list in the Administration.config file. The provider has to be strongly typed and added to the GAC (Global Assembly Cache).
The provider can be added to the trusted assemblies by adding the following line to your <trustedProviders> section in the following file: C:\Windows\System32\inetsrv\Config\administration.config.
<add type=”Microsoft.SharePoint.Administration.Claims.SPClaimsAuthMembershipProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c” />
However, due to a glitch in IIS 7, this will still not allow you to manage the users via IIS. See this post for information on trustedProviders.
Now, you have the issue of trying to manage users in the SQL db. You can access the users that are in the database from sharepoint and grant them permissions, but you cannot actually add roles or users to the SQL db. Here is where I simply point you in the right direction, but have a little less detail. You can, for now, use Visual studio to open the web site that sharepoint created and go to the website menu, then ASP.NET Configuration and create more users/roles as we did with the first one, but this can be a pain for regular use.
Also, this video is pretty short and has some good info on ASP membership, if you are fairly new to it.
Finally, for our site, we made a page with this createuserwizard for user addition and there is definitely a lot that you can do with the basic wizards, but it may help, as most likely if you are using this form of authentication, users will be registering themselves for the site anyway. This wizard is a basic template for registration for your users. Maybe not ideal when we go to production, but it works to et things going quickly for dev. Good luck!
<membership defaultProvider=”SQLMembershipProvider”>
<providers>
<add connectionStringName=”SQLConnectionString” passwordAttemptWindow=”5″ enablePasswordRetrieval=”false” enablePasswordReset=”false” requiresQuestionAndAnswer=”true” applicationName=”/” requiresUniqueEmail=”true” passwordFormat=”Hashed” description=”Stores and Retrieves membership data from SQL Server” name=”SQLMembershipProvider” type=”System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” />
</providers>
</membership>