We encountered the following issue in our SharePoint 2013 installation.
We implemented a content type which inherited from the Document Set content type. In this content type we included a lookup column which pointed to a custom list. This custom list has over 30000 items (List view threshold is 50000 so this is not an issue in our environment.)
When we try to view or edit the properties of the Document set via the UI (DispForm.aspx, EditForm.aspx), an error appears on the screen (“Something went wrong”) and we get the following error in the ULS:
System.InvalidOperationException: Error during serialization or deserialization using the JSON JavaScriptSerializer. The length of the string exceeds the value set on the maxJsonLength property
The error is caused by the the lookup column (pointing to the large list) that SharePoint tries to show when it displays the content type inheriting from Document Set. Strangely this error occurs only in this content type. We have other content types inheriting from the Item CT, which include this very same site column, and behave normally. Perhaps, in the case of the Document Set there is some intermediary step that generates this error, before showing the properties.
Anyway, the workaround for the time being is to stop showing this column in the forms.
